gh-149819: fix .pth and .start file processing in subprocess when inheriting PYTHONPATH#150177
Conversation
After PR pythongh-149583 (Fix double evaluation of .pth and .site files in venvs), .pth files are no longer loaded in subprocesses started with subprocess.run([sys.executable, ...]). The root cause: main() seeds known_paths from removeduppaths() with all sys.path entries inherited from the parent process. addsitedir() then skips .pth processing for every directory already in known_paths. Fix: - main(): call removeduppaths() for dedup but start known_paths as a fresh empty set, so that addsitedir() processes .pth files in every site-packages directory regardless of inherited sys.path. - addsitedir(): move known_paths.add() before the sys.path.append and guard the append with 'sitedir not in sys.path' to avoid creating duplicate entries when called with a fresh known_paths. This preserves the pythongh-75723 dedup guarantee while allowing subprocesses to load .pth files.
* Extend _make_start() and _make_pth() to take an optional `basedir` which is used instead of `site.tmpdir` if given. * Add test_pth_processed_when_sitedir_already_on_path() to test the core GH#149819 bug: .pth files in subprocesses aren't handled if PYTHONPATH pointing to the .pth directory is inherited. * Similarly add test_start_processed_when_sitedir_already_on_path() to verify that .start files in the same circumstances are also now processed.
|
Also, reproducer by @tcely |
|
Also, original bug report by @scoder |
|
This looks good to me. Thanks! |
|
Thanks for working on the site.py issues, BTW. That tends to be hairy business, regularly risking to break the setup of random people in some spot of the globe. I do appreciate it. |
Co-authored-by: scoder <stefan_ml@behnel.de>
And we had few core team members committed to maintaining site.py, but it seems like a natural place for me to get back involved in. Thank you both for engaging on this issue. I'm close to committing the branch. |
|
Thanks @warsaw for the PR 🌮🎉.. I'm working now to backport this PR to: 3.15. |
|
GH-150202 is a backport of this pull request to the 3.15 branch. |
…hen inheriting PYTHONPATH (GH-150177) (#150202) gh-149819: fix .pth and .start file processing in subprocess when inheriting PYTHONPATH (GH-150177) * gh-149819: Fix .pth files not loaded in Python subprocesses After PR gh-149583 (Fix double evaluation of .pth and .site files in venvs), .pth files are no longer loaded in subprocesses started with subprocess.run([sys.executable, ...]). The root cause: main() seeds known_paths from removeduppaths() with all sys.path entries inherited from the parent process. addsitedir() then skips .pth processing for every directory already in known_paths. Fix: - main(): call removeduppaths() for dedup but start known_paths as a fresh empty set, so that addsitedir() processes .pth files in every site-packages directory regardless of inherited sys.path. - addsitedir(): move known_paths.add() before the sys.path.append and guard the append with 'sitedir not in sys.path' to avoid creating duplicate entries when called with a fresh known_paths. This preserves the gh-75723 dedup guarantee while allowing subprocesses to load .pth files. * Fill out the tests for GH#149888 * Extend _make_start() and _make_pth() to take an optional `basedir` which is used instead of `site.tmpdir` if given. * Add test_pth_processed_when_sitedir_already_on_path() to test the core GH#149819 bug: .pth files in subprocesses aren't handled if PYTHONPATH pointing to the .pth directory is inherited. * Similarly add test_start_processed_when_sitedir_already_on_path() to verify that .start files in the same circumstances are also now processed. * Update Lib/site.py * Oops! Remove redundant code --------- (cherry picked from commit 3c298e2) Co-authored-by: Barry Warsaw <barry@python.org> Co-authored-by: BugBounty Mind <bugbounty-mind@deepseek.tui> Co-authored-by: scoder <stefan_ml@behnel.de>
Followup to #149888 and #149819 - Fix by @IntentBug in the original PR, with test cases and review by @warsaw